{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "191ad6b2",
   "metadata": {},
   "source": [
    "## Inception\n",
    "\n",
    "The Inception V1 architecture was an important milestone in the development of CNN classifiers. Before its inception, the common idea to improve Deep Neural Networks was to try and stack more layers, hoping for better performance. The main hallmark of this architecture is the improved utilization of the computing resources inside the network. This was achieved by a carefully crafted design that allows for increasing the depth and width of the network while\n",
    "keeping the computational budget constant. Details can be found at https://arxiv.org/abs/1409.4842v1\n",
    "\n",
    "\n",
    "One of the most straightforward way of improving the performance of deep neural networks is by increasing their size. This includes both increasing the depth: the number of levels – of the network and its width: the number of units at each level.  However blindly increasing the depth and width often leads to overfitting because of the increase in the number of parameters. Additionally, it also increases the computation resources needed to train and deploy the neural network.\n",
    "\n",
    "The Inception network was heavily engineered. It used a lot of tricks to push performance; both in terms of speed and accuracy. There have been several stages of improvement: Inception_v1, Inception_v2, Inception_v3, Inception_ResNet etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9850edc",
   "metadata": {},
   "source": [
    "Here we will consider the main idea behind Inception_V1, the first Inception architecture that started it all. \n",
    "\n",
    "Prior deep learning architectures typically stacked convolutional filters in a sequential manner i.e each layer applies a set of convolutional filters of same size, and passes it on to its subsequent layer.  The kernel size of the filter at each layer depended on the architecture. \n",
    "\n",
    "However, how do we know that we have chosen the right kernel size at each layer? \n",
    "Intuitively, we expect huge variation in the information. Thus, a larger kernel is preferred for information that is distributed more globally, and a smaller kernel is preferred for information that is distributed more locally. By being forced to choose one kernel size, the resulting architecture may not be optimal.  This is the problem that Inception v1 tried to solve.\n",
    "\n",
    "This was done by using an Inception module. The idea is to have wider layers that allowed for multiple filter sizes at the same level. \n",
    "\n",
    "\n",
    "The naive implementation of the Inception module performs convolutions on the input using 3 different kernel sizes (1x1, 3x3, 5x5). Additionally, max pooling is also performed. The outputs are concatenated, and sent into the next inception module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "52365b61",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "94a0a586",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NaiveInceptionModule(nn.Module):\n",
    "    def __init__(self, in_channels, num_features=64):\n",
    "        super(NaiveInceptionModule, self).__init__()\n",
    "        # 1x1 branch\n",
    "        self.branch1x1 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, num_features, kernel_size=1, bias=False),\n",
    "                        nn.BatchNorm2d(num_features, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # 3x3 branch\n",
    "        self.branch3x3 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, num_features, kernel_size=3, padding=1, bias=False),\n",
    "                        nn.BatchNorm2d(num_features, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # 5x5 branch\n",
    "        self.branch5x5 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, num_features, kernel_size=5, padding=2, bias=False),\n",
    "                        nn.BatchNorm2d(num_features, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # Pooling\n",
    "        self.pool = torch.nn.MaxPool2d(kernel_size=3, stride=1, padding=1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        conv1x1 = self.branch1x1(x)\n",
    "        conv3x3 = self.branch3x3(x)\n",
    "        conv5x5 = self.branch5x5(x)\n",
    "        pool_out = self.pool(x)\n",
    "        out = torch.cat([conv1x1, conv3x3, conv5x5, pool_out], 1)\n",
    "        return out\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed58545",
   "metadata": {},
   "source": [
    "This naive inception block has a major flaw. Using even a small number of 5x5 filters can prohibitively increase the number of parameters. \n",
    "This becomes even more expensive when we add the pooling layer. This is because the  number of output filters equals to the number of filters in the previous stage. Thus when we concatenate the output of the pooling layer with the outputs of convolutional layers would lead to an inevitable increase in the number of output features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "08445899",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters: 573824\n"
     ]
    }
   ],
   "source": [
    "naive_inception_module = NaiveInceptionModule(in_channels=256)\n",
    "\n",
    "x = torch.rand((1, 256, 28, 28))\n",
    "out = naive_inception_module(x)\n",
    "\n",
    "# Note that the output has more number of features than the input. \n",
    "assert out.shape == torch.Size([1, 64+64+64+256, 28, 28])\n",
    "\n",
    "num_params = sum(p.numel() for p in naive_inception_module.parameters() if p.requires_grad)\n",
    "print(f\"Number of parameters: {num_params}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bbe1932",
   "metadata": {},
   "source": [
    "How can we fix this problem?\n",
    "We add more 1x1 convolutional layers!!\n",
    "\n",
    "We use extra 1x1 convolutional layers to reduce the number of input channels before the 3x3 and 5x5 filters. This may seem counterintuitive, but 1x1 convs are much cheaper than 3x3 and 5x5. And reducing the input channels drastically reduces the number of parameters of the 3x3 and 5x5 convs. Additionally, 1x1 convolution is also applied after pooling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "755435e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Inceptionv1Module(nn.Module):\n",
    "    def __init__(self, in_channels, num_1x1=64, \n",
    "                 reduce_3x3=96, num_3x3=128, \n",
    "                 reduce_5x5=16, num_5x5=32,\n",
    "                 pool_proj=32):\n",
    "        super(Inceptionv1Module, self).__init__()\n",
    "        # 1x1 branch\n",
    "        self.branch1x1 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, num_1x1, kernel_size=1, bias=False),\n",
    "                        nn.BatchNorm2d(num_1x1, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # 3x3 branch\n",
    "        # 1x1 conv\n",
    "        self.branch3x3_1 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, reduce_3x3, kernel_size=1, bias=False),\n",
    "                        nn.BatchNorm2d(reduce_3x3, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        # 3x3 conv\n",
    "        self.branch3x3_2 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(reduce_3x3, num_3x3, kernel_size=3, padding=1, bias=False),\n",
    "                        nn.BatchNorm2d(num_3x3, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # 5x5 branch\n",
    "        # 1x1 conv\n",
    "        self.branch5x5_1 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(in_channels, reduce_5x5, kernel_size=5, padding=2, bias=False),\n",
    "                        nn.BatchNorm2d(reduce_5x5, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        self.branch5x5_2 = torch.nn.Sequential(\n",
    "                        nn.Conv2d(reduce_5x5, num_5x5, kernel_size=5, padding=2, bias=False),\n",
    "                        nn.BatchNorm2d(num_5x5, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "        \n",
    "        # Pooling\n",
    "        self.pool = torch.nn.Sequential(\n",
    "                        torch.nn.MaxPool2d(kernel_size=3, stride=1, padding=1), # Pool\n",
    "                        nn.Conv2d(in_channels, pool_proj, kernel_size=1, bias=False),\n",
    "                        nn.BatchNorm2d(pool_proj, eps=0.001),\n",
    "                        nn.ReLU(inplace=True))\n",
    "                        \n",
    "        \n",
    "    def forward(self, x):\n",
    "        conv1x1 = self.branch1x1(x)\n",
    "        conv3x3 = self.branch3x3_2(self.branch3x3_1((x)))\n",
    "        conv5x5 = self.branch5x5_2(self.branch5x5_1((x)))\n",
    "        pool_out = self.pool(x)\n",
    "        out = torch.cat([conv1x1, conv3x3, conv5x5, pool_out], 1)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "702e5e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters: 275680\n"
     ]
    }
   ],
   "source": [
    "inception_v1_module = Inceptionv1Module(in_channels=256)\n",
    "\n",
    "x = torch.rand((1, 256, 28, 28))\n",
    "out = inception_v1_module(x)\n",
    "\n",
    "assert out.shape == torch.Size([1, 64+128+32+32, 28, 28])\n",
    "\n",
    "num_params = sum(p.numel() for p in inception_v1_module.parameters() if p.requires_grad)\n",
    "# Notice how we have drastically reduced the number of parameters.\n",
    "print(f\"Number of parameters: {num_params}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bac1fe0",
   "metadata": {},
   "source": [
    "Using the dimension reduced inception module, a neural network architecture was built. This was popularly known as GoogLeNet. \n",
    "\n",
    "GoogLeNet has 9 such inception modules stacked linearly. It is 22 layers deep (27, including the pooling layers). It uses global average pooling at the end of the last inception module. With such a deep network, there is always the problem of the vanishing gradient. To prevent the middle part of the network from “dying out”, the paper introduced two auxiliary classifiers. This was done by applying softmax to the output of two of the intermediate inception modules, and computed an auxiliary loss over the ground truth. The total loss function is a weighted sum of the auxiliary loss and the real loss."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
